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I.  BACKGROUND 


The  degree  to  which  a  fighting  force  is  able  to  effectively  manage  and  utilize  its 
communications  technologies  has  a  direct  impact  on  mission  accomplishment  and 
survivability,  both  in  peacetime  and  war.  In  modern  war,  the  importance  of 
communications  cannot  be  understated.  In  recent  years,  threats  from  nonstate  actors  as 
well  as  failing  nation  states,  emerging  naval  superpowers,  and  natural  disasters,  have 
reinforced  the  need  for  timely  and  accurate  communications  amongst  assets.  The  new 
asymmetric  and  dynamic  worldwide  threat  environment  dictates  that  all  Department  of 
Defense  (DoD)  assets,  not  just  U.S.  naval  assets,  have  access  to  fresh  information  that 
impacts  their  mission.  In  response  to  this  new  environment,  the  U.S.  Navy  has  begun 
shifting  its  warfighting  philosophy  from  one  of  platform-centric  warfare  to  one  of  net- 
centric  warfare.  The  concept  of  net-centric  warfare  envisions  naval  forces  as  a  network  of 
sensors,  weapons  delivery  systems,  and  decision  makers  rather  than  platfonns  (ships, 
aircraft,  submarines)  working  semi-autonomously  toward  a  common  goal.  The  net- 
centric  concept  flattens  hierarchy  and  increases  situational  awareness  of  all  connected 
members,  which  enables  decisions  on  intelligence  that  often  needs  to  be  acted  upon  when 
it  is  only  hours  old.  The  integration  of  the  DoD  communications  and  computer  systems 
into  the  Global  Information  Grid  (GIG)  was  the  backbone  of  the  Navy’s  initiative  toward 
using  the  power  of  information  technology  (IT)  to  increase  the  agility  of  combat  forces 
and  increase  the  speed  and  effectiveness  with  which  the  military  is  deployed  while 
satisfying  the  need  for  resiliency  and  reliability  in  the  face  of  severe  harm.1 

For  submarines,  the  need  to  share  infonnation  in  net-centric  fashion  with  naval 
and  joint  assets  had  exposed  shortfalls  in  submarine  communications  technologies.  In  the 
early  to  mid-1990s,  submarine  communications  suites  consisted  largely  of  older,  legacy 
systems  that  were  not  designed  with  net-centric  capability  in  mind,  or  piecemeal  radio 
suites  involving  partial  integration  of  new  technologies  from  commercial  off-the-shelf 
(COTS)  vendors.  As  a  result,  the  ability  of  any  given  submarine  to  participate  in  net- 

1  Lawrence  Jones  and  Fred  Thompson,  From  Bureaucracy  to  Hyperarchy  in  Netcentric  and  Quick 
Learning  Organizations  (Charlotte,  NC:  Information  Age  Publishing,  2007),  242-243. 
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centric  warfare  varied  considerably.  In  1995,  the  U.S  Navy  decided  it  needed  a  “network¬ 
centric  communications  system  designed  to  support  the  command  and  control 
requirements  of  the  submarine  force,”  and  that  this  new  system  would  “provide  seamless, 
transparent,  secure  connectivity  for  information  exchange  between  submarine  users  and 
other  Joint,  Naval,  Department  of  Defense  (DoD),  Federal,  Allied  and  Coalition  force 

9 

users  of  the  Global  Information  Grid  (GIG)  in  support  of  submarine  warfare  task  areas. ” 
This  system  was  the  Common  Submarine  Radio  Room  (CSRR). 


-  Integrated  Maritime  Communications  Systems  Mission  Need  Statement,  1995,  2. 
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II.  INTRODUCTION 


CSRR  represents  a  paradigm  shift  in  the  way  submarine  communications 
technology  is  procured,  integrated,  and  managed.  CSRR  integrates  existing  program  of 
record  (POR)  technology  with  COTS,  government-off-the-shelf  (GOTS), 
Nondevelopmental  Item  (NDI)  hardware,  and  application-specific  software  through  an 
open  architecture  approach  into  a  common  communications  suite  for  all  submarines.  The 
CSRR  system  functions  as  a  communications  gateway,  and  is  interoperable  with  DoD 

3 

Command,  Control,  Communications,  Computers  and  Intelligence  (C4I)  infrastructure. 
The  goal  for  CSRR  is  to  leverage  existing  Navy  C4I  acquisition  programs  (PORs)  to 
create  a  common  communications  baseline  across  the  submarine  fleet,  not  to  develop 
new  technology.  Thus,  CSRR  is  a  system  of  systems  (SOS).  Commonality  is  attained  via 
standardized  user  and  equipment  interfaces,  managed  by  control  and  management  (C+M) 
software  designed  for  CSRR.  Using  an  open  architecture,  along  with  nonproprietary 
standards/protocols,  means  that  CSRR  is  able  to  rapidly  respond  to  changes  in  equipment 
needs  due  to  mission  or  obsolescence  issues,  yet  leverage  existing  POR  investment, 
which  keeps  the  life  cycle  cost  down. 

It  is  more  than  the  technology  itself,  however,  that  makes  CSRR  stand  out  as 
unique.  In  fact,  the  way  the  program  was  formed,  and  continues  to  be  implemented  and 
managed,  warrants  study  and,  as  such,  will  be  the  focus  of  this  thesis. 


3  Revision  4  of  the  Test  and  Evaluation  Master  Plan  (TEMP)  for  Common  Submarine  Radio  Room 
(CSRR),  2009, 1-1. 
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III.  LITERATURE  REVIEW 


Much  has  been  written  on  management  of  large  government  projects  and  defense 
acquisition  systems  and,  certainly,  general  management  principles  from  these  disciplines 
apply  to  managing  a  program  like  CSRR,  but  that  is  where  the  similarities  end.  Net¬ 
centric  programs  like  CSRR  that  rely  on  the  program  office  to  integrate  existing  and  new 
technologies  for  a  need  that  has  no  expiration  date  present  special  management 
challenges  that  have  just  recently  been  expounded  in  print.  In  their  book,  Organizing  For 
a  Complex  World,  Chao  and  Ben-Ari  write  that  the  management  of  complex  DoD 
systems  has  itself  become  increasingly  complex,  due  to  new  systems  emphasizing  net- 
centric  technology,  systems  of  systems  (SOS)  engineering  approaches,  and  multi-mission 
configurations.4  Unlike  past  Cold  War  era  acquisitions,  maintaining  a  technological  edge 
today  means  adapting  to  the  rapid  pace  of  technological  change  in  the  face  of  a 
consolidating  industrial  base,  limited  budgets,  and  more  challenging  operational 
environments.5  The  field  of  information  technology  (IT)  is  particularly  susceptible  to 
these  factors,  and  adds  to  them  an  increasingly  blurred  demarcation  between  purely 
government  and  purely  civilian  technology.6 

Other  issues  regarding  the  management  of  complex  systems  are  the  questions  of 
who  should  manage  them,  and  how  they  should  be  managed.  Chao  and  Ben-Ari  argue 
that  the  federal  government’s  capability  and  capacity  for  systems  integration  has  declined 
over  the  last  two  decades,  and  this  has  led  to  shortcomings  in  management.7 8  Table  1 
compares  capabilities  of  two  types  of  organizations  (government  and  industry)  as  they 

q 

relate  to  systems  integration.  According  to  Chao  and  Ben-Ari,  the  government  only  has 
an  advantage  in  organizational  longevity  when  compared  to  industry,  and  is  lacking  in 
traits  such  as  project  management  skill  and  technical  awareness. 

4  A.  P.  Chao  and  G.  Ben-Ari,  Organizing  for  a  Complex  World:  Developing  tomorrow's  defense  and 
net-centric  systems  (Washington,  DC:  The  CSIS  Press,  2009),  31. 

5  Chao  and  Ben-Ari,  Organizing  for  a  Complex  World,  1. 

6  Ibid.,  3. 

7  Ibid.,  1. 

8  Ibid.,  64. 
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Table  1.  Comparison  of  Government  and  Industry  Attributes  for  Systems  Integration 

(From  Chao  and  Ben-Ari,  2009). 


Key  attributes  for 
systems  integrator 

Government 

Industry 

Technical  Awareness 

- 

+ 

Project  Management 
Skill 

- 

+ 

Customer 

Understanding 

+/- 

+ 

Organizational 

Longevity 

+ 

- 

Manufacturing 

Expertise 

- 

+ 

Organizational 

Independence 

- 

- 

The  perception  that  industry  can  better  manage  an  acquisition  program  than 
government  is  sometimes  related  to  the  perception  of  government  control  being  too  tight 
and  centralized.  With  programs  like  CSRR,  which  involve  rapidly  changing  technological 
and  mission  needs,  loose  central  control  can  cause  interoperability  problems,  while  tight 
central  control  slows  issue  resolution  and  practically  guarantees  obsolete  systems.9 
Indeed,  over  the  past  three  decades,  the  government’s  performance  in  public  sector 
management  has  been  criticized  as  inefficient,  ineffective,  too  costly,  overly  bureaucratic, 
overburdened  by  unnecessary  rules,  and  failing  in  the  provision  of  either  the  quantity  or 
quality  of  services  deserved  by  the  taxpayers.10 

The  management  of  the  CSRR  program,  however,  is  different  from  the  typical 
government  approach  outlined  by  Chao  and  Ben-Ari  and  discussed  by  Jones  and 
Thompson.  The  CSRR  program  office  works  with  program  offices  from  respective  PORs 
to  oversee  the  integration  of  equipment  interfaces,  yet  they  do  not  have  control  over  the 
design  or  manufacture  of  these  PORs;  thus,  both  loose  and  tight  central  control  exist 


9  Chao  and  Ben-Ari,  Organizing  for  a  Complex  World,  67. 

10  Jones  and  Thompson,  From  Bureaucracy  to  Hyperarchy,  23. 
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within  the  program  to  facilitate  delivery  of  POR  equipment  that  can  immediately  be 
integrated  with  a  CSRR  suite.  The  CSRR  team  also  must  possess  the  technical  awareness 
and  customer  understanding  requisite  for  presiding  over  a  program  that  meets  all  the 
communications  needs  of  the  submarine  force,  unlike  POR  offices,  which  are  typically 
concerned  with  meeting  their  individual  parameter  obligations.  CSRR  is  a  heavily 
government-run  program,  yet  industry  participation  is  retained  in  those  aspects  in  which 
it  has  the  greatest  cost,  schedule,  and  performance  benefits,  ft  is  these  ways  and  others  in 
which  the  CSRR  program  diverges  from  traditional  acquisition  paradigms  that  make  it  an 
interesting  subject  for  analysis. 
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IV.  A  DECISION  BETWEEN  GOVERNMENT  AND  INDUSTRY 


At  the  genesis  of  the  CSRR  program,  procurement  of  the  VIRGINIA  class 

external  communications  system  (ECS)  radio  rooms  had  been  in  progress  for  five  years, 

headed  by  a  dual  industry  team  of  the  Lockheed  Martin  (LM)  and  Electric  Boat  (EB) 

corporations.  The  VIRGINIA  class  ECS  represented  the  future  direction  of  submarine 

force  communications,  encompassing  net-ready  components  and  an  architecture  that  took 

advantage  of  government  off-the-shelf  (GOTS)  and  commercial  off-the-shelf  (COTS) 

technologies.  But  with  all  the  advances  in  VA  ECS,  its  primary  shortcoming  was  that  it 

would  only  be  common  to  the  VA  platform.  The  goal  for  CSRR  was  to  leverage  the 

investment,  technology,  and  methodologies  in  VIRGINIA  ECS  to  establish  a  single  ECS 

architecture  baseline  for  all  classes  of  Navy  submarines.11  CSRR  would  be  an  open 

architecture  system  that  integrated  existing  programs  of  record  (POR)  while  maximizing 

use  of  GOTS/COTS  and  providing  control  and  management  software  (C+M)  to  manage 

POR  physical  components  to  control,  process  and  disseminate  Command,  Control 

Communications,  Computers,  Intelligence,  Surveillance  and  Reconnaissance  (C4ISR) 
12 

information. 

Leveraging  VIRGINIA  new  construction  ECS  to  provide  a  baseline  ECS 
fleetwide  promised  sweeping  advantages.  First,  stovepipe  architecture  across  the  fleet 
would  be  replaced  with  common  architecture  and  software/user  interfaces.  Second, 
utilizing  a  modular  open  system  architecture  (MOSA)  that  maximizes  the  use  of 
GOTS/COTS  components  would  provide  design  flexibility  needed  for  submarine 
communications  when  new  requirements  emerge  or  technology  becomes  obsolete.  The 
use  of  nonproprietary  standards  and  protocols  would  allow  for  more  rapid  insertion  of 
new  technology  as  it  became  available;  making  modernization  a  “push”  system  from 
industry,  vs.  a  “pull”  system  from  government.  It  was  for  these  reasons,  among  others, 
that  the  U.S.  Navy  decided  to  go  forward  with  the  CSRR  concept. 

^  SPAWAR,  Common  Submarine  Radio  Room  (CSRR)  Acquisition  Plan  (AP)fAcquisition  Strategy’ 
(AS),  (2007),  2. 

1-  SPAWAR,  CSRR  Acquisition  Plan,  1. 
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A. 


THE  WAY  FORWARD 


But  with  whom  to  go  forward  with  was  a  key  question.  LM  had  already 
developed  C+M  software  for  VIRGINIA  and  SEAWOLF  platforms,  and  was  in  position 
to  leverage  what  it  had  learned  on  VIRGINIA  ECS.  By  contrast,  the  government  had 
decades  of  in-house  expertise  with  submarine  communications  and  fleet  support,  to 
include  back-fit  integration  and  modernization.  As  with  any  new  acquisition  program, 
the  decision  amongst  alternative  awards  would  be  weighed  by  the  factors  of  cost, 
schedule,  and  perfonnance.  In  2002,  there  were  three  options  considered:14 

1 .  Sole  source  contract  to  the  VIRGINIA  industry  team  (LM/EB) 

2.  Open  Competition 

3.  Government  team  in-house  development 

The  lack  of  available  technical  data  and  schedule  for  development  precluded  an 
open  competitive  award  so,  in  2002,  the  open  competition  option  was  shelved  with  a 
possible  reconsideration  for  LOS  ANGELES  class  submarines  in  the  future.15  With  the 
open  competition  option  removed,  the  only  choice  to  be  made  was  between  industry  and 
government,  which  began  with  an  analysis  of  how  the  industry  team  had  been  performing 
on  VIRGINIA. 

From  1-3  August  2000,  an  Integrated  Baseline  Review  (IBR)  was  conducted  at 
Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems-Eagen  (LMNE&SS)  for  the 
VIRGINIA  class  ECS  program.  The  IBR  team  consisted  of  both  government  and  industry 
members,  and  its  goal  was  to  gain  understanding  of  the  technical  scope  and  time-phased, 
allocated  budget  of  the  work  under  contracts  to  support  VA  ECS.  The  IBR  addressed  all 
industry  efforts  (LM  and  EB)  associated  with  major  review  areas  of  the  component 
development  plan  (CDP)  of  March  1998  but,  due  to  the  phased  nature  of  the  program, 
could  only  address  concerns  with  the  first  phase.  The  team  uncovered  some  disturbing 
findings,  including: 

SPAWAR,  personal  communication,  December  2010. 

14  Ibid.,  and  SPAWAR,  CSRR  Acquisition  Plan,  1 0. 

13  SPAWAR,  personal  communication,  December  2010. 
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1.  Contractor  team  cost  growth  of  approximately  $1.5  million  for  the  first 
phase  of  the  program. 

2.  A  budget  for  logistics  efforts  that  was  based  on  assumptions  of 
deliverables  from  GOTS/COTS  programs  that  were  not  part  of  PORs,  and 
that  these  deliverables  appeared  under-budgeted. 

3.  Potential  lack  of  interoperability  with  the  rest  of  the  Navy  and  potential 
further  cost  growth  due  to  not  incorporating  all  available  GOTS  PORs 
with  VIRGINIA  ECS  PORs. 

In  addition  to  the  IBR,  review  of  a  July  2000  LM  cost  report  by  SPAWAR 
showed  that  LM  had  moved  over  $2  million  from  its  program  management  budget  to 
cover  cost  growth  in  other  areas,  with  an  implied  total  cost  growth  of  over  $5  million 
when  factoring  in  scope  of  work.16  This  finding,  along  with  the  IBR  results,  prompted  a 
recommendation  from  government  to  perform  an  in-depth  review  of  contractor  team  cost 
reports  and  remaining  work  scope  to  define  future  potential  cost  growth.  The  full 
weighing  of  cost,  schedule,  and  performance  risks  still  needed  to  be  done  to  adequately 
compare  both  options,  but  it  seemed  that  the  government  had  good  reason  to  suspect  the 
industry-only  option  would  involve  risk  of  cost  growth. 

During  2002,  multiple  comparisons  of  the  cost,  schedule,  and  performance  risks 
between  the  government  and  industry  options  were  done.  The  NAVSEA  Cost 
Engineering  Team  performed  an  independent  assessment  of  the  two  alternative 
approaches  for  implementing  CSRR  for  SSBN  platforms17  and,  in  February  of  2002, 
PMW  173  conducted  an  analysis  of  alternatives  (AO A)  for  the  three  original  options. 
Table  2  is  the  risk  mitigation  matrix  compiled  by  PMW  173  showing  that  the  government 
option  presented  the  lowest  overall  risk,  with  schedule  and  cost  risks  having  near  equal 
outcomes,  mainly  attributable  to  the  government  having  to  redevelop  the  C+M  software. 


16  Dan  Brothers,  Concern  Report,  #  Program  Management  6,  1. 

17  John  C.  McNellis,  “To  RADM  Phil  Davis,  USN ,”  unpublished  letter  (2002),  1. 
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Sole  source  Contract 

Competitive  Contract 

Government 

Cost  Risk  (ability  to  provide  CSRR  within 
the  current  funding) 

High-  EB/LM  costs  have  consistently  been 
higher  than  the  independent  Navy  estimate 

High-  technical  data  is  not  available;  can’t  be 
a  fixed-price  contract;  include  cost  to 
redevelop  software 

Moderate/High-  labor  costs  are  significantly 
lower;  include  cost  to  redevelop  software 

Technical  Risk  (ability  to  provide  and 
operationally  effective  and  suitable  CSRR  for 
OHIO  and  LA,  with  minimial  differences 
front  the  VA  and  SEAWOLF  variants) 

Moderate-  have  access  to  all  the  technical 
data,  but  have  not  shown  flexibility  to 
accommodate  changes  (even  further 
definition  of  GFE  equipment  has  been  called 
a  change  of  scope) 

High-  technical  data  is  not  available;  starting 
from  ground  zero,  constrained  by  another 
company’s  architecture 

Moderate-  government  involvement  in 
VIRGINIA  means  not  starting  at  ground 
zero;  good  understanding  of  all  the  GFE 
equipment 

Schedule  Risk  (ability  to  perform  per  the 
desired  contract  schedule  I.e,  provide  CSRR 
install  kits  LAW  with  existing  funding) 

High-  EB/LM  hasn’t  been  able  to  meet 
current  schedules 

High-  getting  a  late  start;  need  to  redevelop 
software  from  scratch 

High/Moderate-  redevelop  software  from 
scratch  but  have  control  software 
experience,  possibly  use  IRM  as  basis;  gains 
10-11  months  over  other  approaches. 

Funding  Risk  (ability  to  obligate  and 
execute  funds  IAW  the  existing  funding 
profile) 

Moderate/High-  sole  source  would  be 
shorter  contracting  cycle  than  competitive, 
but  past  EB  negotiations  have  been  lengthly 

High-  time  required  for  RFP,  proposals, 
award  negotiations  will  force  later  start  date. 

Low-  can  release  fimding  to  begin  effort 
months  earlier 

Table  2.  Risk  Assessment  Matrix  (From  PMW  173,  2002). 
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In  conjunction  with  this  risk  assessment,  PMW  173  itemized  cost  comparisons 
between  the  government  option  and  a  possible  govemment/industry  option  to  see  where 
the  largest  cost  differences  and  risks  would  reside.  Table  3  is  the  cost  comparison  chart 
generated  by  PMW  173  for  the  SSBN  CSRR  program.18 


SPAWAR,  personal  communication,  December  2010. 
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Table  3.  Cost  Comparison  for  the  SSBN  CSRR  Options,  (in  thousands  of  dollars) 

(From  PMW  173,2002). 


COST  EFFORT  ORG 
TYPE 


System  NUWC 
Engineering 


LM 

Software  1 


Software 

Convergence  SSC-SD 


Testing  |  NUWC 


Alteration 

Documentati 


Program 

Management 


Certification 
Facility  NUWC 


Installation 

Planning 


Subtotal 


Hardware 

Procurement 


ECSE 

Procurement 

and 

Production  LM 


Fabrication 

and 


Test  Witness, 

Shipping, 

Receipt  Insp.  SSC-CH 


Installations  SSC-CH 


Subtotal 


Total 

(NRE+RE) 


GOV’T 
ONLY  (A) 

GOV’T 

INDUSTRY 

(B) 

DELTA  (A- 
B) 

9,423 

8,300 

1,123 

0 

4,640 

-4,640 

4,065 

1,080 

2,985 

0 

3,360 

-3,360 

2,500 

0 

2,500 

761 

761 

0 

2,482 

2,482 

0 

2,532 

2,927 

0 

693 

693 

0 

2,095 

2,095 

0 

24,551 

26,338 

-1,787 

38,018 

14,306 

23,712 

0 

43,940 

-43,940 

28,130 

25,772 

2,358 

0 

2,358 

-2,358 

18,054 

17,284 

770 

84,202 

103,660 

-19,458 

108,753 

129,998 

-21,245 
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Another  comparison  was  done  for  the  SSGN  CSRR  program,  with  the  column 
DELTA  showing  similar  differences  for  categories  in  direction,  although  with  different 
magnitudes.  There  are  no  fundamental  differences  between  the  SSGN  cost  comparison 
data  and  the  SSBN  data;  what  follows  is  a  discussion  concerning  the  SSBN  data. 

There  were  some  cost  differences  between  the  government  and  the 
government/industry  options  that  are  worthy  of  discussion.  Table  3  shows  that  the 
government/industry  approach  would  be  overall  $3,517  million  more  expensive  for 
systems  engineering,  yet  software  development  for  the  two  options  would  be  nearly  equal 
in  cost.  The  most  notable  differences  between  the  two  options  occur  for  recurring  costs 
(RE),  in  which  the  largest  industry  contribution  was  procurement  and  production  of  the 
LM  ECSE  racks.  The  differences  in  cost  comparisons  between  the  government  and 
government/industry  options  are  predicated  on,  among  other  things,  the  respective  roles 
that  LM  and  the  government  would  have  in  developing  CSRR  for  the  SSBN  platform. 
Concerns  about  the  ability  of  either  option  to  mitigate  potential  high  cost  and  schedule 
risks  were  present  in  statements  from  either  side.  The  independent  assessment  by  PMW 
173  prompted  a  response  from  Lockheed  Martin’s  president  about  the  “apparently 
unknowable  basis  of  estimate  for  the  cost  of  the  SPAWAR/NUWC  solution  in 
comparison  with  the  fully  justified  industry  costs  put  forth  by  Lockheed  Martin.”19  In  one 
instance,  LM  felt  that  the  $4.6  million  estimate  for  software  development  was  too  low, 
and  suggested  that  LM  as  the  software  design  agent  receive  a  Cost  Plus  Fixed  Fee 
(CPFF)  contract,  with  the  cost  likely  being  closer  to  $10  million.  But  the  important 
differences  between  the  two  options  went  beyond  just  cost;  the  overall  goal  of 
establishing  common  hardware  and  software  at  minimum  cost  while  maintaining  the 
current  SCN  funded  schedule  for  SSGN  conversion"  shaped  what  roles  the  government 
and  LM  would  play  when  contracts  would  be  awarded.  As  time  progressed,  and  the 
industry-only  option  became  less  attractive,  analysis  was  done  for  a  govemment/industry 
option,  with  LM  still  playing  a  chief  role  in  hardware  development  and  design.  Below  are 


19  McNellis,  “To  RADM  Davis,”  1. 

20  SPAWAR,  CSRR  Acquisition  Plan,  9. 
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brief  summaries  of  each  solution,  taken  from  both  SPAWAR  and  LM  perspectives,  to 
include  advantages,  disadvantages,  and  differences  in  proposals. 

B.  THE  GOVERNMENT-ONLY  OPTION 

Because  the  government  would  have  to  redevelop  C+M  software  for  CSRR, 
commonality  across  VIRGINIA,  SEAWOLF,  and  SSBN/SSGN  platforms  would  be 
reached  on  a  later  timeline.  This  option  would,  therefore,  present  some  technical  and  cost 
risks  with  establishing  commonality  as  a  backfit  to  VIRGINIA  and  SEAWOLF.  For 
hardware,  the  government  option  utilized  modified  Langley  racks  for  COTS  equipment, 
which  saved  money  when  compared  to  the  industry  option  ECSE  racks."  Costs  for 
systems  engineering  and  labor  would  be  lower  due  to  lower  staff-year  costs,  and  this 
option  presented  a  10  month  schedule  advantage  due  to  the  ability  to  immediately  begin 
obligating  funds."  The  schedule  advantage  was  very  important  in  order  to  avoid 
disrupting  SSBN  deployment  and  availability  rotations.  Another  concern  was  that  CSRR 
would  integrate  many  existing  PORs  that  were  already  under  the  cognizance  of  the 
government.  CSRR  was  going  to  be  80%  government-furnished  equipment  (GFE)  or 
government-furnished  information  (GFI),  so  independent  of  the  approach,  the  liability  for 

24 

providing  a  large  portion  of  the  equipment  would  reside  with  the  government. 
Efficiencies  with  GFI/GFE  could  therefore  be  realized  with  an  all-government  approach. 
By  comparison,  on  industry  run  VIRGINIA/SEA  WOLF  ECS  over  560  GFE/GFI  items 
had  to  be  issued  twice  by  the  government;  once  for  VIRGINIA,  and  once  for 
SEAWOLF.25  History  of  the  trident  IRRs  also  showed  that  total  ownership  costs  (TOC) 
were  lowered  when  utilizing  existing  government  configurations  and  facilities;  the  annual 
cost  to  maintain  a  Trident  IRR  configuration  model  at  LM  was  $3.5  million,  but 
relocating  it  to  a  govermnent  facility  lowered  the  annual  cost  to  only  $1  million.26  The 

-1  SPAWAR,  personal  communication,  December  2010. 

22  Ibid. 

22  SPAWAR,  CSRR  Acquisition  Plan,  10. 

-4  SPAWAR,  personal  communication,  December  2010. 

25  Ibid. 

26  Ibid. 
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last  advantage  to  consider  was  that  the  human  capital  for  submarine  communications 
expertise  had  overwhelmingly  resided  in  the  government  for  decades. 


C.  THE  GOVERNMENT/INDUSTRY  OPTION 

The  government/industry  option  presented  a  better  technical  solution,  with  both 
parties  concentrating  on  their  respective  core  competencies.  LM  would  be  the  software 
design  agent,  and  the  government  would  control  integration,  assembly  and  testing.”  For 
costs  concerns,  LM  suggested  that  the  government  re-evaluate  the  use  of  the  Q-70  ECSE 
racks  for  use  with  COTS,  in  order  to  leverage  development  investment  and  reduce  total 
life  cycle  cost,  while  using  the  in-place  Langley  racks  for  GOTS  equipment.  Leveraging 
the  LM  C+M  software  already  developed  for  VIRGINIA  and  SEA  WOLF  would  give  the 
fleet  an  earlier  common  software  baseline,  while  minimizing  the  effects  on  VIRGINIA 
and  SEAWOLF  programs.  The  government/industry  approach  would,  therefore, 
mitigate  some  cost  and  schedule  risk  by  allowing  development  of  the  OHIO  CSRR  in 
parallel  with  VIRGINIA  and  SEAWOLF  CSRR.30  The  major  disadvantage  for  the 
govemment/industry  option  was  the  increased  NRE  and  RE  costs  due  to  using  LM 
hardware;  SPAWAR  predicted  additional  RE  and  NRE  of  over  $24  million  and 
$3  million,  respectively,31  for  the  SSBN/SSGN  platforms. 

D.  THE  CONTRACT  DECISION 

The  final  decision  made  by  PMW  770  was  to  use  the  government/industry  option, 
although  in  a  slightly  different  form  than  discussed  above.  LM  was  awarded  a  sole  source 
Cost  Plus  Award  Fee  (CPAF)  contract,  number  N00039-03-C-0026,  in  March  of  2003  to 
develop  the  OHIO  CSRR  C+M  software.  ”  LM  was  justified  as  the  only  responsible 
source  for  the  work  based  on  their  previous  VIRGINIA  and  SEAWOLF  software  efforts. 


22  SPAWAR,  personal  communication,  December  2010. 

28  McNellis.  “To  RADM  Davis,”  1. 

22  SPAWAR,  CSRR  Acquisition  Plan,  6. 

30  SPAWAR,  CSRR  Acquisition  Plan,  7,  and  SPAWAR,  personal  communication,  December  2010. 
3 '  SPAWAR,  personal  communication,  December  2010. 

32  SPAWAR,  CSRR  Acquisition  Plan,  9. 
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The  LM  ECSE  racks  were  not  to  be  used  to  support  COTS  equipment;  instead,  modified 
Langley  racks  would  be  used.  NUWC  NPT  would  perfonn  the  system  integration  efforts, 
and  SSC-CH  would  oversee  the  production  efforts. 

The  govermnent/industry  option  presented  the  earliest  and  best  technical  solution 
for  the  submarine  force,  saving  money  through  lower  government  labor  costs  while 
gaining  a  10  month  schedule  advantage  (by  being  able  to  immediately  obligate  funds  and 
start  work),  while  providing  the  in-house  expertise  needed  to  flex  the  program  as 
necessary.  This  solution  also  allowed  CSRR  to  be  fielded  on  the  SSGN  platforms 
without  adversely  affecting  their  conversion  schedules. 


33  SPAWAR.  CSRR)  Acquisition  Plan,  10. 
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V.  TRAINING:  THE  MULTI-RECONFIGURABLE  TRAINING 

SYSTEM 


A  significant  advantage  of  the  CSRR  program  is  the  development  of  the  Multi- 
Reconfigurable  Training  System  (MRTS),  which  enables  the  fleet  to  meet  greater 
flexibility  requirements  for  effective  communications  training,  while  providing  this 
flexibility  at  a  lower  cost  per  unit  than  legacy  radio  room  trainers.  For  years,  OHIO  class 
sailors  trained  on  shore  using  integrated  radio  room  (IRR)  trainers.  IRR  trainers  were 
proprietary,  hardware-intensive,  expensive  to  groom  (maintain  through  periodic 
maintenance),  and  were  not  upgraded  often.  Furthermore,  the  IRR  trainers  could  only  be 
used  for  OHIO  class  crews.  LOS  ANGELES  crews  often  did  not  have  separate  external 
communications  system  (ECS)  trainers,  resulting  in  training  being  conducted  on 
shipboard  tactical  equipment,  an  evolution  often  not  feasible  in  port.  The  lack  of 
available  quality  training  on  fast  attack  submarines  contributed  to  significant  deficiencies 
in  submarine  communications  capability  in  recent  years.  The  SSBN  fleet,  on  the  other 
hand,  had  regular  training  opportunities,  but  on  radio  room  technology  that  had  not  been 
updated  in  almost  20  years.  The  current  pace  of  fleet  C4I  upgrades  can  no  longer  afford  a 
radio  room  that  goes  unchanged  for  two  decades. 

MRTS  gives  the  fleet  the  flexibility  to  fix  these  issues  via  a  design  that  utilizes 
COTS  hardware  and  a  common  software  baseline.  A  MRTS  trainer  is  a  series  of 
flatscreen  panels  run  by  control  software  that  can  mimic  shipboard  component  interfaces, 
to  include  providing  high-definition  graphic  simulations  of  CSRR  system  hardware 
circuit  emulation.34  MRTS  can,  therefore,  be  used  for  operator,  maintenance  and  team 
training,  requiring  the  student  to  operate  the  system  as  they  would  at  sea  while  recording 
every  student  action  for  instructor  feedback.  The  control  software  is  common  to  a 
baseline,  yet  provides  functionality  for  loading  different  variants:  the  result  is  a  trainer 
that  can  run  multiple  radio  room  configurations  on  the  same  hardware.  Unlike  IRR, 
MRTS  allows  any  trainer  to  be  used  by  OHIO,  LOS  ANGELES,  VIRGINIA, 
SEAWOLF,  and  SSGN  platforms  simply  by  loading  different  variant  software,  which 

34  SPAWAR,  personal  communication,  December  2010. 
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35 

usually  takes  only  an  hour.  This  allows  training  commands  to  schedule  an  OHIO  crew 
in  the  morning  and  a  LOS  ANGELES  crew  after  lunchtime  (for  instance),  in  the  same 
trainer.  MRTS  has,  therefore,  both  increased  available  training  opportunities  for  fast 
attack  crews  while  mitigating  schedule  demands  for  regular  rotation  of  SSBN  and  SSGN 
prepatrol  training  periods. 

Remarkably,  the  increase  in  training  flexibility  offered  by  MRTS  has  also  come  at 
reduced  costs  and  with  an  increased  speed  of  delivery  relative  to  legacy  technology. 
Procuring  a  baseline  hardware  trainer  for  CSRR  would  cost  approximately  $22  million 
per  trainer,  compared  to  $500,000  when  using  COTS  hardware  components.36 
Maintaining  a  baseline  hardware  configuration  for  MRTS  would  require  expensive 
grooming  and  maintenance,  and  upgrading  the  baseline  would  cost  between  $2  million 
and  $6  million  for  each  trainer.37  Using  COTS  hardware  allows  the  common  baseline  to 
be  managed  through  software;  therefore,  the  cost  of  baseline  upgrades  is  lower  and  can 
also  be  spread  across  multiple  sites.  Leveraging  these  cost  savings  has  allowed  the 
Navy  to  procure  eleven  MRTS  trainers  for  what  two  IRR  trainers  would  have  cost,  with 
baseline  upgrades  costing  less  than  $  1  million  per  trainer.  This  represents  a  potential  cost 
avoidance  of  up  to  $26.5  million  per  trainer,  or  $291  million  across  the  entire  program, 
for  procurement  and  initial  baseline  upgrade. 


33  Personal  communication  with  Bangor  Trident  Training  Facility,  September  2010. 

36  SPAWAR,  personal  communication,  December  2010. 

37  Ibid. 

3^  Ibid. 
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VI.  RISK  MITIGATION  AND  PROGRAM  DESIGN 


In  2006,  Deputy  Secretary  of  Defense  Gordon  England  and  Undersecretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics  Ken  Krieg  both  remarked  that  the 
ability  of  the  government  to  manage  rising  complexity  in  weapons  and  defense  systems 
programs  was  one  of  the  most  important  challenges  facing  the  Pentagon.  As  reported  in 
Ben-Ari  and  Chao,  defense  researcher  Robert  Dietrick  has  defined  “complexity”  as  it 
pertains  to  weapons  and  other  defense  systems  as  “the  number  of  interactions  and  degree 
of  integration  amongst  subsystems,  as  well  as  the  degree  of  integration  at  the  component 
level.”40  Dietrick  also  suggested  that  increased  complexity  and,  therefore,  increased 
functionality  and  capability,  can  adversely  affect  a  program’s  cost,  schedule,  and 
performance  outcomes.  Popular  examples  supporting  Dietrick’s  point  are  the  Joint  Strike 
Fighter  (JCS),  Future  Combat  System  (FCS),  and  DDG-1000  Zumwalt  acquisition 
programs,  each  of  which  has  experienced  significant  cost  and/or  schedule  overruns,  and 
possesses  a  high  degree  of  technical  complexity.41 

The  need  for  modern  systems  to  have  multi-mission  and  multi-function 
capabilities,  as  well  as  to  integrate  net-ready  technology,  has  made  management  of 
modem  acquisition  programs  correspondingly  more  complex.42  This  challenge  is 
uniquely  acute  with  SOS  programs  like  CSRR,  in  which  many  distinct  technologies  and 
systems  are  linked  through  a  common  data  network.  Most  SOS  programs  have  an 
indefinite  lifetime  due  to  a  continual  need  for  the  capability  (like  submarine 
communications),  and  thus  the  challenges  of  integrating  new  technology  with  old, 
managing  installation  and  testing  schedules,  and  ensuring  fleet  interoperability  will  never 
go  away43  for  these  programs. 


39  Chao  and  Ben-Ari,  Organizing  for  a  Complex  World,  xiii. 

40  Ibid.,  33. 

41  Ibid. 

42  Ibid.,  31. 

43  Ibid.,  69. 
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Despite  the  obstacles  that  complex  SOS  technologies  pose  to  the  program 
manager,  however,  evidence  abounds  of  successful  complex  programs  delivered  on  time 
and  on  cost.  For  example,  the  VIRGINIA  class  production  timeline  was  able  to  be 
shortened  despite  systems  onboard  becoming  more  complex;  this  contradicts  Dietrick’s 
statement,  and  suggests  that  negative  cost,  schedule,  and  performance  outcomes  of  an 
acquisition  program  are  independent  of  the  system’s  complexity.  One  can  reconcile  this 
contradiction  by  realizing  that  complexity  does  not  necessarily  equal  risk,  and  that 
managing  a  system’s  risk,  not  a  system’s  complexity,  most  clearly  delineates  the 
respective  roles  and  obligations  of  technical  leadership  and  government.44  Government 
must  be  comfortable  that  risks  created  by  complexity  are  foreseeable,  manageable,  and 
acceptable,45  and  put  management  processes  in  place  to  mitigate  these  risks. 

Risk  management  is  done  to  a  certain  extent  for  all  acquisition  programs  but,  as 
stated  earlier,  more  complex  systems  pose  unique  challenges  that  tend  to  require  more 
rigorous  risk  management  and  oversight.  It  is  often  organizations  with  the  most  robust, 
clearly  defined,  and  formally  documented  risk  management  practices  that  succeed  when 
others  fail,  or  that  sustain  the  least  amount  of  damage  when  events  outside  their  control 
threaten  success.  It  is,  therefore,  important  to  analyze  risk  mitigation  practices  of 
complex  programs  like  CSRR  to  gain  lessons  learned  that  can  be  applied  to  other 
complex  programs.  Risk  mitigation  is  a  principle  and,  therefore,  is  scalable  to  the  needs 
of  different  programs  and  stakeholders,  so  the  basic  tenets  derived  from  comparing  and 
contrasting  risk  mitigation  practices  of  different  programs  are  worthwhile.  It  is  in  this 
context  that  risk  mitigation  in  the  CSRR  program  will  be  compared  to  results  of  a  task 
force  assessment  involving  platform  and  weapons  certification  in  the  surface  warfare 
community. 


44  Chao  and  Ben-Ari,  Organizing  for  a  Complex  World,  15. 

45  Ibid. 
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A.  SURFACE  FLEET  CERTIFICATION 

A  1998  Chief  of  Naval  Operations  (CNO)  message  directed  NAVSEA  06  (now 
NAVSEA  05)  to  implement  a  process  that  coordinates  installations  and  testing  of  fleet 
systems’  interoperability  earlier  in  the  inter-deployment  cycle.46  In  2004,  policy  and 
guidance  regarding  C5I  modernization  was  released  by  Commander,  U.S  Fleet  Forces 
Command.  In  2005,  the  Naval  Warfare  Systems  Certification  Policy  (NWSCP)  was 
written,  and  since  has  been  the  governing  document  for  assessment  and  certification  of 
warfare  systems  for  U.S.  Navy  surface  warfare  platfonns.  The  NWSCP  is  used  by  surface 
fleet  leadership  to  support  warfare  systems  installation  decisions  and  to  assess  readiness 
for  operational  deployment.  NWSCP  focuses  on  three  major  events:  the  Certification 
Readiness  Review  (CRR),  the  Initial  Platfonn  Certification  Decision  (IPCD),  and  the 
Platform  Certification  Decision  (PCD).  Of  the  three,  PCD  provides  the  final  platform 
level  certification  prior  to  deployment.  The  PCD  is  typically  conducted  after  a  ships  final 
predeployment  maintenance  period  and  after  shipboard  testing  of  associated  warfare 
systems.  The  intent  of  the  PCD  is  to  ensure  that  no  system  degradation  has  occurred 
between  the  IPCD  and  the  PCD  that  will  affect  operational  capabilities.  The  current 
NWSCP  was  intended  to  be  the  first  phase  of  a  three-phase  policy  development,  but 
currently  no  further  policy  has  been  promulgated. 

In  recent  years,  many  surface  warfare  platfonns  experienced  problems  relating  to 
the  NWSCP  warfare  and  platform  certification  process,  ranging  from  the  acceptance  of 
degraded  combat  system  capability  to  delayed  deployments.47  Among  these  cases  was  the 
platform  certification  of  USS  STERETT  (DDG  104)  that  occurred  in  June  of  2010. 

NAVSEA  05  conducted  USS  STERETT  (DDG  104)  platfonn  certification 
decision  (PCD)  on  03  June  20 10. 48  The  purpose  of  the  PCD  was  to  evaluate  the  DDG  104 
baseline,  which  consisted  of  19  POR  warfare  systems,  some  of  which  are  common  to 
CSRR.  The  evaluation  covered  software,  firmware,  hardware,  documentation,  and 

46  SPAWAR,  personal  communication,  December  2010. 

47  Ibid. 

48  NAVSEA  Washington,  DC,  USS  STERETT  (DDG  104)  Platform  Certification  Decision  (PCD), 

(2010),  2. 
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training  in  order  to  assess  the  maturity  and  readiness  of  the  platform  for  deployment. 
Although  DDG  104  did  not  fully  satisfy  all  the  criteria  for  the  PCD  (there  are  17  criteria 
in  the  NWSCP  common  to  IPCD  and  PCD),  it  was  certified  for  deployment  with 
operational  considerations/limitations  resolved  via  system  level  documentation,  and  4 
fleet  advisories  in  effect.  Among  the  criterion  not  satisfied  were: 

Criterion  1:  Warfare  system  computer  programs  have  no  unresolved  high  priority 
or  safety  trouble  reports  (TRS):  DDG  104  was  certified  with  four  warfare  systems  not 
fully  meeting  this  criteria.  Issues  with  these  warfare  systems  were  reported  “resolved 
with  system  level  documentation,”  that  is,  procedural  workarounds  and/or  documented 
limitations  of  the  systems  were  put  into  place. 

Criterion  2:  Successfully  pass  a  25-hour  stress  and  endurance  test:  The 
interoperability  certification  committee  (ICC)  identified  a  “significant  stability  issue” 
unique  to  the  DDG  104  platform,  however  certification  was  granted  in  part  due  to  the 
testimony  of  CO  USS  STERETT,  who  indicated  that  the  stability  issues  were  within  his 
ability  to  operationally  manage. 

Criterion  6:  Further  platform  interoperability  (I/O)  testing:  DDG  104  was  certified 
for  deployment  with  “C2  capabilities  consistent  with  known  documented  issues.” 
NAVSEA  did  remark  that  the  baseline  under  review  was  a  significant  improvement  over 
the  old  one,  and  represented  the  “highest  interoperability  performance  when  compared  to 
other  platforms  in  the  fleet  today.”  Interoperability  issues  included  degraded  ID  reliability 
and  engagement  issues,  and  possible  degradation  of  situational  awareness  and  confidence 
in  the  automated  ID  process. 

The  USS  STERETT  PCD  case  is  just  one  that  exemplifies  fleet  stakeholder 
concerns  regarding  the  certification  of  surface  warfare  systems  and  platforms.  Among 
these  concerns  is  that  certification  criteria  is  not  rigidly  defined,  testimony  from  ships 
personnel  can  be  used  in  place  of  more  objective  evidence,  and  that  the  overall  process 
invokes  too  much  variability,  and  thus  repeatability  and  unifonnity  in  the  fleet  suffers. 
Certain  people  felt  that  some  factors  lead  to  certifications  being  “rubber  stamped”  if 
systems  were  not  totally  broken,  due  to  a  lack  of  funds  and  time  to  fix  issues  prior  to 
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deployment.49  Other  people  did  not  approve  of  the  basic  principle  of  resolving  issues  by 
defining  limitations  through  documentation  and  multiple  workarounds,  as  this  knowledge 
sometimes  did  not  get  integrated  at  the  deckplate  level. 

B.  CORRECTIVE  ACTION:  THE  NAVAL  WARFARE  SYSTEMS 

CERTIFICATION  TASK  FORCE 

The  DDG  104  certification  and  other  events  influenced  the  formation  of  the  Naval 
Warfare  Systems  Certification  Task  Force  (NWSCTF)  in  August  of  20 10. 50  The  task 
force  was  headed  by  RADM  Thomas  G.  Wears,  Commander,  Naval  Undersea  Warfare 
Center  (NUWC)  with  the  goal  of  conducting  a  coordinated,  comprehensive  assessment  of 
the  state  of  Naval  Warfare  Systems  certification  processes.  The  task  force  focused  on  the 
following  areas  (summarized): 

1.  The  process  for  establishing  and  resourcing  modernization  requirements 
for  C5ISR  systems. 

2.  The  combat  systems  certification  process,  and  its  effectiveness  in 
supporting  follow-on  platform  certification  events. 

3.  The  effectiveness  of  C5ISR  interoperability  systems  engineering  and 
testing,  including  evaluation  of  the  PCD  timeline. 

4.  The  PCD  process  to  include  certification  requirements,  assessment 
metrics,  risk  management  practices,  fleet  involvement,  and  previous 
platform  certification  events  which  indicated  significant  issues  limiting 
full  or  any  level  of  certification  for  a  platform.  This  category  included 
issues  relating  to  safety  and  readiness. 

5.  Manpower  as  it  relates  to  the  conduct  of  the  Warfare  Systems  Certification 
Process 


49  SPAWAR,  personal  communication,  December  2010. 

50  Commander,  Naval  Sea  Systems  Command.  Establishment  of  the  Naval  Warfare  Systems 
Certification  Task  Force,  (2010),  1. 
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6.  The  technical  authority  structure  necessary  to  support  warfare  systems 
certification  across  SYSCOMS,  warfare  centers,  and  program  offices. 

These  categories  were  divided  into  five  pillars  amongst  the  task  force:  Combat 
System  Certification,  Platfonn  Certification,  C5I  Modernization  and  Resourcing, 
Interoperability,  and  Technical  Authority  Structure.  The  task  force  Platfonn  Certification 
Pillar  produced  38  major  and  15  minor  findings,  which  are  grouped  into  the  13  high-level 
findings  in  Table  4.  Findings  from  other  pillars  significant  to  this  thesis  are  summarized 
in  Table  5.  Many  of  the  high-level  findings  contained  multiple  parts,  so  the  brief 
description/example  column  in  Table  4  reflects  the  portion  of  the  finding  significant  to 
existing  CSRR  practices. 
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Table  4.  High-Level  Findings  Summary  (From  NWSCTF,  2010). 


High-Level  Finding 

Brief  description/example 

Right  criteria  at  warfare  system  cert  level 
are  not  clearly  identified/articulated 

No  clear  definition  of  which  or  how 
many  criteria  must  be  acceptable  to 
determine  a  no-cert  status.  Thresholds 
are  debated  at  certification  panels.  Other 
definitions  of  cert  criteria  are  not  clear. 

No  non-conformance  process  to  allow 
acceptance  of  risk  by  proper  authorities 

Current  practice  is  debate  amongst  panel 
members  as  to  what  is  “yellow”  vs. 

“red,”  often  without  technical  authority 
present.  No  consistent  requirement  for 
documentation/implementation  of  non¬ 
conformance  approval  process. 

Inconsistent  defect  prioritization  and  risk 
standards. 

No  single  source  data  repository  for 
trouble  reports 

Timeline  for  insta  1/cert  events  is 
insufficient  for  providing  decision  makers 
with  an  off-ramp  if  there  is  an  issue. 

Current  policy  places  certification 
decision  at  deployment  minus  1  month. 

Documentation  of  capabilities  and 
limitations  is  inadequate 

Integration  at  the  console  level  is  left  to 
the  sailor,  who  must  ready  many 
documents,  interpret  engineering 
terminology,  determine  which  items  are 
applicable  to  their  jobs,  and  then 
remember  them,  even  in  stressful 
situations. 

Interoperability  evaluations  are  not 
conducted  early  enough  to  improve  the 
outcome 

Testing  conducted  late  in  timeline,  test 
sites  not  sufficiently  used  to  produce 
results  in  timely  manner. 

(Major  finding)  Existing  model  permits 
installations  on  multiple  platforms  before 
OT  complete  on  first  platform. 
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Table  5.  Significant  Findings  From  Other  NWSCTF  Pillars  (From  NWSCTF,  2010). 


Pillar 

Finding 

Brief 

de  s  c  riptio  n/e  x  a  mple 

C5I  Modernization 

Systems  installed  that  have 
not  completed  OT,  but  were 
given  authorization  via  LRIP 
fromPEO’s. 

Problem  is 
exacerbated  by 
insufficient  integrated 
testing  during 
development.  Ships 
have  been  delivered  to 
the  fleet  not  ready  for 
tasking. 

C5I  Modernization 

There  are  numerous  C5I 
baselines  throughout  the  fleet. 

Causes  interoperability 
concerns,  version 
control  concerns,  and 
does  not  effectively 
mitigate  ILS  cost  risk. 

Combat  System 

Lack  of  “right  people,  right 
place,  right  time” 

Lack  of  involvement  of 
appropriate  fleet 
leadership  early  in  the 
development  process. 
Technical  authority 
representation  is  not 
sufficient. 

Interoperability 

Net- Ready  KPP’s  are 
insufficient. 

Insufficiently  detailed, 
measurable,  and 
testable. 

What  follows  is  a  line-by-line  analysis  of  these  issues  as  they  relate  to  the  CSRR 
program  practices  and  methods  of  risk  mitigation.  To  normalize  the  analysis  and, 
therefore,  make  the  findings  as  comparable  to  the  CSRR  program  as  possible,  findings 
from  the  system  acceptance  test  (SAT)  phase  of  the  CSRR  development,  integration,  and 
testing  timeline  were  selected  for  comparison.  Like  PCD,  SAT  “locks  in”  a  baseline  that 
typically  remains  unchanged  until  the  next  baseline  is  approved.  SAT  is  the  third  and 
final  test  event  for  a  CSRR  baseline,  which  follows  Engineering  Change  Proposal  (ECP) 
development  testing  and  System  Design  Verification  Testing  (SDVT).  SAT  also 
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performs  the  function  of  verifying  that  the  CSRR  baseline  under  test  satisfies  all  lower- 
level  system/sub  system  specifications,  as  well  as  higher  level  requirements  set  forth  by 
the  program  office.51  Where  applicable  to  the  NWSCTF  findings,  elements  of  CSRR 
SDVT  and  other  program  attributes  (including  the  TEMP  and  CSRR  process  and  product 
quality  assurance  plan)  will  be  discussed.  The  specific  SAT  used  for  comparison  is  for 
the  SSBN  1.1. 3.0  baseline  report  from  27  March  2009. 

The  Right  Criteria  at  the  warfare  system  certification  level  are  not  clearly 
identified  and  articulated.  This  finding  deals  mainly  with  the  issues  of  definition  and 
documentation.  NWSCTF  found  that  definitions  of  what  constituted  an  acceptable  test 
criterion,  and  how  many  unacceptable  criteria  could  be  allowed  before  the  entire 
certification  was  unacceptable,  were  lacking.  As  a  result  of  unclear  definitions,  thresholds 
for  pass/fail  criteria  are  debated  at  certification  panels. 

The  CSRR  baseline  had  to  satisfy  criteria  clearly  delineated  in  six  governing 
documents  when  undergoing  SAT,  to  include  strategic  communications  parameters, 
connectivity  requirements,  KPPs  and  System  Data  Exchange  Matrix  circuit  timing 
requirements,  requirements  not  satisfied  during  SDVT  (if  applicable),  system  level  test 
plans  for  certain  message  types,  and  a  stress  test  plan  for  target  change  messages.  These 
documents  contain  clear  definitions  of  what  constitutes  a  successful  test  from  the 
individual  circuit  level  to  the  overall  system  level.  Because  CSRR  is  a  system  that 
integrates  PORs  and  legacy  systems,  testing  of  CSRR  capabilities  necessarily 
incorporates  acceptance  criteria  for  its  subsystems.  The  SAT  documentation  also  contains 
detailed  definitions  of  the  pass/fail  criteria,  which  covers  everything  from  the  KB  size  of 
message  to  be  transmitted,  to  the  quality  of  data  transferred. 

There  is  no  nonconformance  process  to  allow  acceptance  of  risk  by  proper 
authorities.  The  current  practice  is  a  debate  among  panel  members  of  standards  and 
thresholds,  including  what  “red”  and  “yellow”  standards  should  be.  A  requirement  for  a 
nonconformance  approval  process  is  not  available  at  the  element,  combat  system,  or 
warfare  system  level. 

Naval  Warfare  Center  Division  Newport,  Rhode  Island.  CSRR  1 . 1 .3 .0  Systems  Acceptance  Test 
(SAT)  Report,  final  version,  (2009). 
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Testing  documentation  for  CSRR  baselines  contain  thorough,  unambiguous,  and 
detailed  pass/fail  criteria  for  each  system,  subsystem,  and  parameter  being  tested. 

The  CSRR  program  is  set  up  in  a  manner  that  attacks  and  resolves 
nonconformance  issues  early.  The  CSRR  PM  works  closely  with  PMs  from  respective 
PORs  during  the  design  phase  to  resolve  interoperability  issues  before  any  testing  takes 
place.  The  CSRR  Product  Quality  Assurance  Plan  (PPQA)  is  a  comprehensive  document 
that  is  used  to  delineate  the  roles  and  responsibilities  of  technical  authority  and  project 
leadership,  as  well  as  outline  mitigation  measures  that  decrease  nonconformity  issues  at 
all  project  stages.  Testing  of  a  complete  CSRR  baseline  may  begin  as  early  as  19 

C  "5 

months  prior  to  the  first  baseline  installation,  in  order  to  work  out  issues  early.  These 
are  a  few  examples  of  entrenched  risk  mitigation  measures  that  help  the  CSRR  process 
adapt  to  changing  fleet  schedules  and  needs,  and  allow  quick  assessment  of  technology 
maturation  when  these  factors  rapidly  change. 

One  CSRR  example  of  nonconformance  risk  being  accepted  by  the  proper 
authorities  was  the  interim  fielding  decision  of  the  increment  1  version  2  (INC1V2) 
CSRR  kits  on  SSN  21,  SSGN  726,  and  SSGN  729,  which  occurred  on  24  September 
20 10. 54  Authorized  fielding  of  these  kits  had  been  given  pending  follow-on  operational 
testing  (OT),  but  the  SSGN  schedule  precluded  the  OT  from  happening,  and  threatened  to 
delay  it  by  almost  a  year.  A  delay  in  the  OT  for  this  increment  would  have  postponed 
delivering  the  capability  for  two  SEAWOLF  and  two  SSGN  platforms  by  two  to  three 
years.  Thankfully,  the  robust  and  early  testing  schedule  of  CSRR  baselines  and 
increments  provided  sufficient  evidence  to  allow  fielding  this  increment  without  the 
subsequent  OT.  The  decision  was  given  to  the  Milestone  Decision  Authority  (MDA), 
PEO  C4I,  which  concluded  that  “extensive  and  successful  testing”  of  the  INC1V2 
upgrade  had  been  conducted,  with  results  that  “substantiate  the  operational  integrity  of 

5-  SPAWAR,  Common  Submarine  Radio  Room  (CSRR)  Process  and  Product  Quality’  Assurance  Plan, 

1-1. 

53  SPAWAR,  personal  communication,  December  2010. 

54  Program  Executive  Officer,  Command,  Control,  Communications,  Computers  and  Intelligence. 
Interim  Fielding  of  Common  Submarine  Radio  Room  (CSRR)  Increment  One  Version  Two  (INC1V2)  Kits 
on  SSN  21,  SSGN  726,  and  SSGN  729  Acquisition  Decision  Memorandum,  (2010),  1. 
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CSRR  INC1V2  and  minimal  risk  associated  with  fielding”  prior  to  the  OT.  To  give  an 
example  of  what  “extensive”  is  with  regard  to  CSRR  testing,  the  1. 1.3.0  baseline  version 
was  able  to  satisfy  88%  of  its  overall  requirements  during  SDVT,  and  by  the  end  of  SAT 
had  satisfied  98%  of  its  overall  requirements.  Additionally,  no  CSRR  baseline  or 
increment  upgrade  is  allowed  to  field  with  any  priority  1  or  2  (significant)  problems,  and 
all  priority  3  problems  require  workarounds  to  be  documented  at  the  system  level  and 
tracked  in  a  central  database. 

There  are  inconsistent  defect  prioritization  and  risk  standards,  and 
inadequate  training  related  to  these  risks  for  those  affected  parties.  Contained  within 
this  finding  is  that  there  is  no  single  source  data  repository  for  trouble  reports. 

The  CSRR  program  utilizes  the  CIMS  database  to  generate,  track,  and  modify 
trouble  reports  found  during  testing  phases.  This  is  a  single  integrated  database 
containing  all  trouble  reports  that  is  accessible  by  all  key  stakeholders. 

The  timeline  for  installation  and  certification  events  is  insufficient  for 
providing  decision  makers  with  an  off-ramp  if  there  is  an  issue.  The  current  policy  for 
the  surface  fleet  places  the  IPCD  at  one  month  prior  to  availability,  and  the  certification 
decision  at  deployment  minus  one  month.  This  creates  schedule  risk,  as  it  makes  changes 
difficult  to  make  prior  to  installation  activities  beginning.  Additionally,  technical 
commands  and  operational  authorities  do  not  consistently  share  schedule  change 
information,  which  affects  planning,  readiness  assessment,  and  certification  events. 

As  stated  earlier,  system  testing  for  a  CSRR  baseline  can  occur  up  to  19  months 
prior  to  the  first  installation  of  that  baseline.  The  CSRR  PM  works  closely  with  fleet 
operational  leadership  to  coordinate  installation  and  testing  of  new  increments  and/or 
baselines  with  the  availability  schedules  of  submarines  such  that  they  will  not  negatively 
affect  the  deployment  cycle.  Part  of  being  able  to  mitigate  schedule  risks  with  the 
installation  is  also  the  rigorous  and  comprehensive  testing  that  all  CSRR  systems 
undergo.  CSRR  is  unique  in  that  it  is  the  only  fleet  C4I  system  that  undergoes  SOS 
testing  prior  to  installation;  the  strength  of  SOS  testing  specific  to  CSRR  is  that 
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interoperability  and  interface  issues  are  able  to  be  fixed  in  the  lab  months  prior  to 
installation  and,  therefore,  without  operational  impact  to  submarine  deployment 
schedules. 

SOS  testing  and  knowing  availability  schedules  are  just  a  couple  of  ways  that  the 
CSRR  team  mitigates  schedule  risk,  the  other  involves  an  understanding  of  what  other 
work  is  to  be  performed  during  these  availabilities,  so  that  the  CSRR  team  can  identify 
windows  of  opportunity  to  get  ahead  of  schedule  and  save  the  Navy  money.  For  the  POM 
2010  process,  the  CSRR  team  outlined  an  alternative  plan  for  688  class  submarine  CSRR 
integration  that  would  save  the  Navy  time  and  money.55  The  team  realized  that  688  class 
submarines  were  scheduled  to  receive  ECS  upgrades,  beginning  in  FY12,  which  would 
incorporate  considerable  hardware,  cabling,  and  infrastructure  changes.  Submarines  of 
the  688  class  were  also  scheduled  to  begin  CSRR  installations  in  FYs  15-18,  so  the  initial 
approach  created  inefficiencies  due  to  component  interfaces  needing  to  be 
designed/installed  twice.  Their  solution  was  to  accelerate  CSRR  for  688  class  submarines 
to  FY12  in  order  to  align  it  with  component  modernization  plans;  conducting  some 
architecture  and  infrastructure  that  would  be  needed  later  on  for  full-blown  CSRR,  while 
giving  the  submarines  new  ECS  capability  as  originally  planned  in  FY12.  Components 
that  were  already  scheduled  to  be  upgraded  would  be  done  in  a  manner  consistent  with 
CSRR  requirements  in  order  to  avoid  duplicate  efforts,  thus  reducing  the  scope  of  the 
final  CSRR  installation  and  further  mitigating  cost  and  schedule  risks. 

Documentation  of  capabilities  and  limitations  is  inadequate.  Among  the 
concerns  expressed  in  this  finding  is  that  there  is  no  single  consolidated  document  that 
ships’  force  can  reference  to  understand  limitations  and  workarounds.  The  operating 
principles  and  procedures  (OPPs)  and  quick  reference  guides  (QRGs)  are  written  for 
specific  systems  and  do  not  include  any  workarounds.  This  leaves  integration  at  the 
console  level  to  the  Sailor,  who  must  reference  many  documents,  interpret  engineering 
tenninology,  and  remember  it,  even  in  stressful  situations. 


55  SPAWAR,  personal  communication,  December  2010. 
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With  the  advent  of  CSRR  came  a  paradigm  shift  in  the  way  technical  and 
operational  documentation  was  organized  for  submarine  radio  rooms.  Troubleshooting 
and  operating  procedures  are  now  written  at  the  communications  circuit  level,  not  the 
component  level.  For  instance,  the  old  IRR  way  of  troubleshooting  was  to  consult 
manuals  for  the  individual  components  involved  in  that  circuit  (like  VLF  or  EHF).  These 
manuals  seldom  took  into  account  the  interface  and  interoperable  nature  of  components 
in  the  radio  room,  and  instead  focused  in  internal  hardware  or  software  faults.  The  new 
way  presents  a  consolidated  and  integrated  view  of  the  room,  where  an  operator  who  is 
having  problems  transmitting  EHF  references  procedures  that  include  troubleshooting 
interfaces  between  subsystem  components.  Workarounds  for  CSRR  can  also  be 
documented  this  way,  as  the  operations  manual  reflects  operation  of  the  circuits,  not  the 
“boxes”  that  make  up  the  circuits.  This  defragments  operational  and  technical  knowledge 
for  Sailors  who  do  not  have  time  to  memorize  engineering  jargon. 

Interoperability  evaluations  are  not  conducted  early  enough  in  the  process  to 
improve  the  outcome.  Currently,  interoperability  testing  is  conducted  so  late  in  the 
timeline  that  it  can  only  effectively  be  used  to  characterize  limitations  to  capability,  or  to 
provide  OQE  to  not  perfonn  the  installation  or  certification  for  deployment. 

CSRR  testing  for  interface  management  and  interoperability  is  inherent  in  the 
testing  program,  which  can  occur  19  months  prior  to  installation.  Using  baseline  1.1. 3.0 
as  an  example,  lab  testing  can  satisfy  98%  of  total  requirements  including  interoperability 
concerns  (due  to  conducting  SOS  testing)  prior  to  installation.  Additionally,  integration 
of  a  new  baseline  or  block  upgrade  is  not  authorized  if  it  potentially  could  adversely 
affect  submarine  deployment  schedules. 

The  existing  C5IMP  model  allows  combat  systems  installations  on  multiple 
platforms  before  combat  systems  certification,  platform  certification,  or  OT  are 
complete  on  the  first  platform.  This  finding  represents  a  concern  over  the  net  result  of 
the  overall  process. 

By  contrast,  CSRR  baselines  are  only  installed  and  tested  on  one  platfonn  before 
going  forward  with  more  installations,  and  this  is  the  same  for  block  upgrades.  The 
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interim  decision  to  field  INC1V2  prior  to  OT  illustrates  this  point;  INC1V2  had  been 
installed  on  only  one  SSGN,  which  was  to  be  the  OT  platfonn. 

Systems  are  installed  that  have  not  successfully  completed  OT  but  were  given 
authorization  via  LRIP  from  PEOs.  As  illustrated  from  the  INC1V2  example,  LRIP  is 
not  used  as  justification  to  install  block  upgrades  to  CSRR  baselines  prior  to  completion 
of  OT.  A  comprehensive  and  robust  integration  and  testing  program  that  satisfies  almost 
all  required  parameters  and  interoperability  concerns  prior  to  installation  was  used  as 
justification  for  installation  without  follow-on  OT. 

There  are  numerous  C5I  baseline  variants  throughout  the  fleet.  The  finding 
did  not  quantify  “numerous,”  but,  nevertheless,  for  operational  and  cost  concerns,  there 
has  been  a  push  from  top  Navy  leadership  to  reduce  the  number  of  baselines  in  the  fleet. 
Instead  of  naval  programs  fighting  over  budgets,  top  Navy  leaders  would  prefer  to  fix  the 
inefficiencies  in  current  ship  programs  to  help  generate  funds  for  future  ships.56  In  2007, 
the  Navy  owned  277  ships  but  kept  551  different  engines  in  inventory.  The  Navy  also,  as 
of  2007,  had  57  submarines  but  an  inventory  of  over  161  periscopes  and  masts.  The 
Navy’s  inventory  of  components  in  2007  also  included  7,325  different  types  of  motors, 
36,979  types  of  valves,  and  443  categories  of  generators.  VADM  Paul  Sullivan,  head  of 
Naval  Sea  Systems  Command,  remarked  that  this  has  “been  a  problem  for  20  years.”  The 
Navy  can  no  longer  afford  to  keep  these  kinds  of  inventories,  which  are  largely  held  over 
from  Cold  War  thinking  with  regard  to  ship  design. 

CSRR  has  only  four  components  that  are  specific  to  any  given  baseline,  the 
workstation,  NMT,  ADNS  INC  3,  and  crypto  equipment.  This  means  that  using  CSRR, 
the  entire  fleet  of  submarines  could  be  managed  with  a  maximum  of  16  baselines,  but  the 
goal  is  to  manage  the  fleet  with  much  less.  This  is  formidable  considering  that  CSRR 
integrates  literally  hundreds  of  subcomponents  into  a  common  framework,  and  that  each 
variant  must  be  (for  communication  and  crypto  concerns)  capable  of  communicating  with 
all  other  variants. 


56  Sandra  I.  Erwin,  “Future  Fleet:  Inefficient  Shipbuilding  Jeopardizes  Navy’s  Expansion  Goals, 
(2007),  1. 
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Lack  of  “right  people,  right  place,  right  time.”  The  lack  of  involvement  of 
appropriate  fleet  personnel  early  in  the  process  leads  to  relevant  issues  and  requirements 
not  being  reflected  in  the  development  and  certification  process  of  elements. 

By  contrast,  the  CSRR  PM  works  early  and  closely  with  offices  of  respective 
PORs  during  each  stage  of  development  to  ensure  that  relevant  issues  and  requirements 
are  clearly  established  and/or  resolved  prior  to  system  integration.  The  certification 
process  of  CSRR  as  a  system  involves  testing  of  all  subsystem  POR  interfaces  and 
interoperability  concerns. 

Net-Ready  KPPs  are  insufficient.  The  task  force  found  that  KPPs  were 
insufficiently  detailed,  measurable,  and  testable. 

Net-Ready  KPPs  in  the  CSRR  program  are  unambiguously  detailed  in  the 
developmental  and  testing  documentation;  this  is  a  result  of  the  CSRR  PM  office  working 
with  POR  offices  early  during  the  development  stages,  and  via  an  integrated,  robust 
testing  plan  that  satisfies  all  subcomponent  parameters  relevant  to  CSRR. 
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VII.  CONCLUSION  AND  SUGGESTIONS  FOR  FUTURE 

RESEARCH 


The  CSRR  program  is  unique  in  its  design  and  implementation  in  many  ways  that 
would  enable  success  for  any  program.  The  working  relationships  between  program 
offices,  along  with  comprehensive,  thorough  risk  mitigation  practices  and  robust 
testing/integration  schedules  are  all  solid  practices  that  could  be  scaled  to  other 
acquisition  programs.  This  thesis  has  analyzed  many  of  the  principles,  practices,  and 
processes  of  CSRR  from  the  perspective  of  the  end  user,  the  United  States  Navy,  as  they 
relate  to  cost,  schedule,  and  performance  of  the  program.  Major  findings  of  the  thesis  are: 

1.  The  CSRR  team  has  risk  mitigation,  management,  and  institutional 
practices  in  place  that  appear  to  address  all  the  major  findings  in  the 
NWSCTF  investigation. 

2.  The  MRTS  system  has  a  potential  cost  savings  of  $291  million  across 
initial  installation  and  first  baseline  upgrade  when  compared  to  legacy 
technology.  A  MRTS  trainer  allows  more  flexible  training  for  a  greater 
number  of  submarine  types,  at  a  cost  of  almost  one-sixth  of  an  IRR  trainer. 

3.  The  extent  of  government  in-house  control  over  almost  all  aspects  of  the 
CSRR  program  is  nearly  total;  industry  effort  is  largely  regulated  to 
development  of  software.  Government  control  over  the  program  is 
centralized  to  control  quality  and  continuity,  yet  decentralized  to  increase 
speed  of  delivery  and  encourage  innovation. 

With  these  principles  in  place,  one  overarching  question  remains:  Is  this  program 
successful,  and  by  what  standard  of  measures?  Further  research  in  CSRR  should  focus  on 
more  objective  metrics  utilized  to  judge  the  execution  of  an  acquisition  program  as  it 
relates  to  cost,  schedule,  and  perfonnance.  Such  metrics  should  include: 

1.  Analysis  of  the  financial  management/budget  information  for  CSRR. 
Because  CSRR  integrates  many  PORs  that  have  separate  budgets 
themselves,  financial  information  regarding  CSRR  is  spread  across  many 
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budget  and  financial  documents.  This  analysis  could  focus  on  what  the 
“total”  cost  of  CSRR  is  to  the  fleet,  as  well  as  standard  financial  metrics 
such  as  meeting  budget  targets. 

2.  Analysis  of  Earned  Value  Management  (EVM)  data  to  ascertain  if  the 
CSRR  program  is  meeting  purposed  schedule  demands. 

3.  Trend  analysis  of  testing  different  variants  and  baselines  of  CSRR 
technology  as  they  progress  over  time.  Analysis  of  potential  learning 
curve  trends  and/or  its  correspondence  with  changes  in  the  testing 
instructions/ executions . 

4.  Analysis  of  the  CSRR  model  as  it  would  apply  to  surface  ships. 

5.  Analysis  of  cost  savings  with  respect  to  the  reduced  logistics  burden 
CSRR  presents  when  compared  to  legacy  technology. 
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